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'Were is the wisdom we have lost in knowledge? 
Where is the knowledge we have lost in information?" 

T.S. Eliot, Fourth chorus from •The Rock". 

"La Nature est un temple ou de vivants piliers 
Laissent parfois sortir de confuses paroles; 
I/homme y passent k travers des forets de symboles 
Qui Tobservent avec des regards familiers." 

Baudelaire, "Correspondences", from "Les Fleurs du Mai". 



Introduction 

In this paper we describe our work in the design and construction of 
virtual environments (VEs) from the standpoint of semiotic theory. This is a 
rather novel exercise for us, coming as we do from "mainstream" cognitive 
psychology. However, we have been struck by the value of using semiotics as a 
framework for the guidance and analysis of our work. After all, both 
Baudelaire, in the above citation, and Cunningham (1992) suggest that a 
person's communication with the natural world is unavoidably mediated 
through signs which sometimes introduces confusion into communication with 
and about the natural world. Our belief, explicated in what follows, that well- 
constructed virtual worlds can create in a person the same feelings and 
cognitions that arise from being in the natural world implies that interactions 
with VEs are likewise mediated through signs and are sometimes confusing. 

We begin by explaining what we mean by a VE and describe how we are 
using VEs in education. We propose a theoretical framework that is a first 
attempt to reconcile our work to semiotic theory. We then look at how a group 
of grade seven children selected the signs to represent objects and events in 
four VEs they constructed. This is followed by a discussion of how the sense of 
"presence* engendered by VEs can be heightened through the judicious 
selection and use of icons and symbols. ("Presence" is the sensation the 
participant has of being in another place while visiting a VE.) Finally, we 
speculate about how the manipulation of signs by learners can lead to guidance 
on learning in the absence of real or virtual coaches, 

A virtual environment is generated from sights, sounds and, in the not- 
too-distant future, physical sensations created by a computer frota a 
database. It is an immersive environment that completely surrounds the 
participant in which sights and sounds, presented by stereo optical and 
auditory displays, are stable and locatable in three-dimensional space. To 
create an immersive VE, four conditions must be met. A participant's field of 
view must be wide enough so that objects in the world can be detected by 
peripheral and well as foveal vision. Ideally, this requires optical systems in 
head-mounted displays (HMDs) to subtend a visual angle of 200 degrees 
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horizontally and 120 degrees vertically (Furness, 1989). The position of the 
participant's head and at least one hand must be tracked with six degrees of 
freedom - position on the X, Y and Z axes in the virtual space, and pitch, roll 
and yaw. The participant must be able to communicate with the VE through 
natural behaviors, such as looking, speaking and pointing, and not through an 
interface that uses a keyboard, mouse, joystick, touch screen, or other such 
device. This requires the use of transducers, such as gloves, wands, speech 
recognizers, gesture recognizers, that communicate participant behavior to 
the computer. In a well-designed VE, the participant is not aware of using an 
interface at all (M. Bricken, 1991). Finally, there should be negligible delays in 
the rate at which the virtual environment is updated in response to 
participants' movements and actions. Any noticeable lag between a 
" participant's actions and ensiling changes in the VE's appearance causes 
confusion and, in a few cases, mild nausea referred to as "simulator sickness". 

The work we describe in this paper uses technology that meets these 
four criteria, with the possible exception of the last one. Certain viewpoints in 
some of our VEs present the participant with more visual detail than others. 
When this is the case, the increased demands on the rendering software cause 
the rate at which the environment is redrawn to slow down. However,we have 
not found this to be a serious impediment to learning nor to our research. 

These requirements for a VE are stringent. Others (Heim, 1993) give a 
much broader definition of VEs to include non-immersive environments. 
However, "desktop VR* (Lavroff, 1992) fails to provide first-person (Clancey, 
1993) learning experiences and certainly does not engender presence. While 
non-immersive VEs are certainly effective for some kinds of learning (see 
Kozma et al., 1993) and are certainly more affordable than immersive 
systems, our work does not yet include them. 

The Human Interface Technology Laboratory (HITL) at the University 
of Washington uses VEs in two ways for education. As might be expected, we 
build VEs for specific instructional purposes and see whether students learn 
about particular content when they visit them. For example, Byrne (Byrne, 
Fumess & Winn, 1995) built a VE for grade eleven Chemistry students that 
allowed the construction of atoms and molecules by assembling them in the 
correct configurations. Byrne found that students learned the Chemistry 
content best if they interacted with the VE rather than just observing the 
construction process. 

We also use the construction of VEs by students as a means for them to 
learn content. In fact, this is the technique we use most frequently since 
constructing VEs for school curriculum is very labor intensive which currently 
precludes the production of enough worlds to meet more than just one or two 
curriculum goals. We have found that students can learn world-building skills 
very rapidly (Bricken & Byrne, 1993) and have used the technique to teach 
AIDS awareness (Byrne, 1994) to develop cognitive and motor skills in children 
with neurological probblems (Osberg, 1994a) and to help students learn about 
the biological cycles operating in a wetlands ecosystem (Osberg, 1994b and 
later in this paper.) At the time of writing (April, 1995), we are actively building 
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VEs with over 400 children in 13 elementary, middle and high schools in the 
Puget Sound region of Washington. 



Conceptual Framework: Semiotics and Virtual Envrionments 

The conceptual framework that underlies our research on educational 
uses of VEs draws from a number of sources. From cognitive psychology 
comes our basic conception of how people learn and use what they have 
learned. Our experimental and instructional techniques are influenced by more 
recent ideas of knowledge construction and the situated and social nature of 
learning. The design of VEs draws guidance from research in human factors 
and from the general premises of semiotics. 

Drawing these together, we propose that learning in. VEs arises from 
interactions between participants and objects in VEs at four sequential levels. 
These are Data, Information, Knowledge and Wisdom. Each interaction 
transforms material from the current level to the next. Thus, interaction with 
data produces information; interaction with information produces knowledge; 
and interaction with knowledge produces wisdom. Semiotic theory enters into 
the picture mostly in the first two of these transformations. As we shall 
shortly see, the creation of information from data requires the creation or 
selection of signs, which can be accomplished by world designers, by 
participants themselves or by computer-driven transducers. The creation of 
knowledge from information involves the use of signs by learners whether as 
sources of knowledge or, internalized, as "tools for thought" (Salomon, 1979, 
1988) that lead to knowledge construction. The development of wisdom from 
knowledge is usually not something that can be designed for and takes place 
over a long period, often an entire lifetime. 

Generally, semiotic theory rests on the proposition that we cannot know 
the world as it truly is but only through signs (Cunningham, 1992; Driscoll, 
1990; Driscoll & Lebow, 1992). One reason for this, that is important in the 
construction of VEs, is that people's inability to detect light beyond the visible 
spectrum, sound beyond the range of normal hearing, and so on, means that we 
can only ever be exposed to a relatively constricted "bandwidth" of data from 
the natural world. As Nagel (1974) has pointed out, we can never really know 
what it is like to be a bat even though we may be able to describe how 
echolocation works, transpose a bat's "sonar" down into the audible range and 
study its flight in slow motion. It follows, say semioticians, that knowledge has 
no objective existence independent of the knower and that our understanding of 
the world is therefore personal, inferential and context-dependent. 

Learning begins when signs are created that stand for experience. The 
processes through which data are transformed into information are more 
physiological than cognitive and may therefore be characterised as "pre- 
attentive* (Arbib & Hanson, 1987; Marr, 1982). Moreover, we can exercise no 
willful control over them (Pylyshyn, 1984). This also implies that in our 
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conceptual framework data and information can only be potentially 
meaningful, not intrinsically so (see also Owen, 1985; Duong, 1994). 

Information is data that have acquired structure. The natural world 
provides data about all its features and behaviors. Data are transmitted in 
many ways, as radiation in the electromagnetic spectrum, as sound through 
the propagation of waves through solids, liquids and gases and mechanically as 
texture, mass, temperature or viscosity. Because of the limitations imposed by 
our senses on our ability to receive these data, information can only be 
obtained about a small subset of these data and therefore is not a complete, 
true or objective mapping of the natural world. The first step in the acquisition 
of structure by data to become information is therefore a biologically imposed 
selection of those data that can be detected by the human senses (see also 
Maturana & Varela, 1987). 

Our perceptual systems impose organization on detected data and this 
too occurs preattentively (Treisman, 1988; Marr, 1982). The structures that 
emerge as data are organized following by and large the principles of Gestalt 
psychology (Wertheimer, 1938) both for visual (Pomerantz, 1986; Pomerantz, 
Pristach and Carson, 1989) and auditory (Handel, 1989) perception. 
Information emerges from both the synthesis and analysis of data to allow us 
to discriminate one object from another and to configure objects into spatial 
and temporal relations (Winn, 1994). Thus, for example, following a series of 
actions in the visual pathway, described by Marr (1982), the visual scene is 
organized from edges and blobs into patterns and into discernible and 
potentially recognizable objects. The auditory stream is segmented (Bregman, 
1990) by pitch and rhythm into patterns .that are potentially recogni ^able as 
speech, musical motifs, or other sounds. 

Both the selection of data and their organization are determined by the 
nature of human perception which operates preattentively. Therefore, even 
before we can begin to think about the information our senses present us with, 
that information is already an abstraction from data about the natural world. 
Our knowledge of the world is therefore based on representations of it not 
directly on the data it provides. We can only experience signs for temporally 
and spatially discriminated objects and the configurations that arise from their 
interrelationships. The same is also true of our perception of and knowledge 
about virtual environments. The same perceptual processes operate when we 
see and hear objects and events from inside an HMD as when we see and hear 
things in the natural world. 

However, there are three important differences between creating 
information from data in natural and virtual environments. The first is that 
information is created in the natural world through selection by our senses 
from far more data than can possibly be detected and structured. In a VE, 
everything has to be added. One starts with the "void" (W. Bricken, 1991), adds 
objects and determines their behavior. This means that VEs are designed and 
that the signs found in them have to be selected or made by somebody. 
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The second difference between natural and virtual environments is that 
the signs that represent objects and events are entirely computer-generated. 
This means they can appear in any form and may behave in any manner 
whatsoever. For example, a fixed object may move away from you as you walk 
towards it rather than appearing to get closer. An object may stay suspended 
in mid air when you "drop" it. 

The third difference is that a VE can easily present information from 
data that people cannot usually detect. Because a VE is created from a digital 
database, it can present information about any data that can be expressed in 
a digital format. Today, this means just about anything. Data that normally 
escape the human senses can be captured by sensors and converted by 
transducers into signs that humans can perceive and examine. A marine 
depth-sounder bounces sounds I cannot hear off the sea-bottom I cannot see 
and creates either a graph (iconic sign) or numerical readout in fathoms 
(symbolic sign) that represents the depth and contour of the bottom. The 
transduction process (Winn, 1993) vastly expands the data from which VEs 
can create informational signs. 

These three points remind us that, however strong a sense of "presence* 
it may engender, a VE is not the real thing. The creation of signs from data in a 
VE is therefore a two-step process. First, information designers (who could well 
be, and in our work usually are, the students/participants themselves) create 
signs that hold the information they wish to place in the VE. Second, when the 
participant enters the VE, these signs function as data from which the 
participant will select and find information. For example, in Byrne's 
"Chemistry World* (Byrne, Furness & Winn, 1995), students build atoms and 
molecules by putting virtual protons, neutrons and electrons together by hand. 
The Chemistry curriculum largely determined what information should be 
presented. The signs through which information, such as subatomic particles, 
were to be shown required design decisions that drew on graphic conventions 
(protons are positive and therefore are best shown by a 3-D "plus" sign), 
pedagogy (how should Hgns behave when a student makes a mistake?) and 
human factors (how big should each object appear to be so that it seems 
natural to pick it up and carry it about?) In the four student-built worlds that 
represented wetlands ecology, described more fully below, the students 
themselves faced similar design decisions. Interestingly, it was easy for them 
to decide how to represent objects that could be shown as iconic signs, such as 
animals and plants. However, information that required transduction of data 
and representation as symbols, such as free nitrogen or energy, required a 
great deal of discussion that never led to entirely satisfactory resolution. In 
both worlds, there is no guarantee, of course, that participants will attend to 
the information in the signs they encounter in the VE any more than you can 
guarantee they will notice certain things as they look around in the natural 
world. 

In our conceptual framework, knowledge is constructed from information 
by students. We therefore subscribe to many of the tenets of "constructivism" 
put forth by those who advocate such approaches for educational technology 
(Duffy & Jonassen, 1992; Duffy, Lowyck & Jonassen, 1993). This means that, 
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as far as possible, we want students to select their own signs in virtual 
environments and to come to know the world in terms that are meaningful to 
them, not in terms that are given by a teacher or world designer. This 
approach often gives rise to surprises. For instance, while discussing what 
objects to place in a VE they were building that embodied the principles and 
actions of the carbon dioxide cycle, a group of grade seven students had to deal 
with one group member who wanted to have a basketball in the VE- The 
objects the other students proposed were the expected animals and plants and 
the group as a whole was hostile to the suggestion. However, the student's 
reasoning was impeccable: Basketball players are energetic, breathe in a lot of 
oxygen and exhale a lot of carbon dioxide; when they blow up basketballs, they 
fill them with carbon dioxide. For this student, the basketball represented the 
way respiration changes oxygen into carbon dioxide and also represented the 
sports context in which the process was meaningful for him. (The group 
process being democratic, this student's suggestion was voted down by the 
others and no basketball appeared in the VE!) 

When students build VEs, "construction* takes on additional 
significance. Perhaps we are close to what Harel and Papert (1993) call 
"constructionism" rather than "constructivism*. At the same time as they are 
constructing knowledge, the students are also constructing "physical" 
environments in which that knowledge is put to use. They can construct 
anything from a wetlands ecosystem to an atom or molecule, from a tree to a 
brain. They can make any event occur in their VE that they want, whether it 
mimics scientifically accepted principles of the natural world or something 
entirely from their own imaginations. They can create iconic signs that 
represent familiar objects and are likely to be recognized easily by others; or 
they can invent symbolic signs that form part of a framing metaphor they 
have chosen for their VE. Finally, they are constructing environments in which 
they control behaviors of chosen objects by interacting with them physically - 
picking them up, carrying them to a new place, placing them in interaction 
with other objects. Knowledge construction follows from the phsycial 
construction of an environment that functions as the student wants it to in a 
direct, visceral manner. 

Finally, knowledge becomes wisdom when it is used effectively. We agree 
with Sternberg (1990, p.4) that "wisdom is a virtue providing a compelling 
guide to action.* Unless knowledge is used, it is "inert* (Cognition and 
Technology Group at Vanderbilt, 1990), disconnected both from a meaningful 
context when it was learned and from a context in which it can be wisely 
applied. We believe that VEs have the same potential for "anchoring* 
instruction in contexts that are meaningful to students like the "Jasper* video 
adventures, designed to teach math by the Cognition and Technology Group at 
Vanderbilt (1992), or many of the exercises using "Bubble Dialogue*, developed 
by McMann & 0*Neil (1993) to develop communication and social skills. 
However, it would be audacious to claim that the VEs our students have 
developed have enabled them to use their knowledge with wisdom. The common 
perception of wisdom, and recent studies of its development (Csrikszentmihalyi 
& Rathunde, 1990; Labouvie-Vief, 1990), demonstrate that it develops slowly 
over time. Our research has not been able to study either students working 
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with VEs over extended periods of time nor the consequences of that activity 
over a lifetime. 



World Design: The Kellogg Middle School Project. 

A VE is a metaphorical representation of either physical reality, an 
abstract representation of some aspect of reality, or a combination of both. 
Since semiotics is based on the individuals' use of signs (metaphorical 
representations of some aspect of the individuals knowledge), a VE, as a 
teaching and learning tool, fits well within the semiotic framework. This is 
especially true in the case where virtual learning environments are created by 
the students themselves. At Kellogg Middle School in Seattle, Washington, we 
had the opportunity to undertake such a world-building activity, and based our 
process on semiotic-centered practices. 

The 120 students in the mixed 7th and 8th grade classes were used to 
the constructivist learning paradigm, since all 5 teachers involved in the 
project studied and employed constructivist techniques on a regular basis. 
The focus of the classroom was on personal inquiry within a larger subject 
domain. Our Wetlands project fit within this same model. 

As part of their fall curriculum, these two classrooms were to study 
wetlands ecology. The teachers and HIT Lab staff chose this portion of the 
curriculum as likely to be suitable for a world-building activity. When we 
arrived, the students were very excited about the prospect of developing a 
virtual learning environment as part of their wetlands sequence. We asked a 
great deal of the students; to learn about wetlands ecology, to study a 
particular aspect of the wetlands ecosystem in detail and to develop a virtual 
environment that represented their understanding of that aspect, to learn how 
to model in three dimensions on the computer, and to become, in essence, 
instructional designers. 

Creating a virtual environment is a four-step process: Planning, 
Building, Prog rammin g, and Experiencing. It is during the Planning and 
Building stages that we were able to see just how students were selecting the 
icons, indexes, and symbols (Houser, 1987) for their virtual environment. 

By the time we started the planning stage, students had a fair 
understanding about what the basic contents of a wetland might be. In 
discussing the objects that the students wanted to include, there were many 
that were suggested that would have been considered relatively obvious: 
cattails, frogs, turtles, water, an alligator, the sun, clouds, etc. Planning 
became particularly interesting when the students abandoned their pre- 
conceived notions of what should be in a wetland, and looked beneath the 
surface, both figuratively and in actuality. In addition to the student who 
wanted to include a basketball, one student wanted to look for fossilized 
remains in the peat bog that formed the wetland. Another wanted to include a 
car that had gone off the road and was rusting in the pond, and would build an 
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entire story about how the car had swerved off the road and landed there. And 
another student wanted to build a condominium development, since that is 
what we do to wetlands. It was clear from these examples that some of the 
students were able to bring a great deal of their personal experience into their 
representation of a wetland. TTiis illustrates both Burner's (1990) point of view 
that individuals* personal stories are important in meaning-making, and, more 
generally, Cunningham's (1992) claim that knowledge and knower cannot be 
separated. 

Words, pictures, bodily movement and the like can become part of our 
"Umwelt* (von Uexkull, 1982) - our personal, internalized view of the world - 
even though they may mark objects that have no literal basis in the physical 
world (i.e. culture). The fact that humans can utilize signs that are arbitrary 
and need have no existence in their immediate experience is what makes 
thought possible and distinctly human. Knowledge, therefore does not consist 
of objects or entities that we acquire. Knowledge is better thought of as 
knowing, or as a process. Through our experience in the world, we build ways 
of knowing, struct ores that determine our current understanding. And via 
these structures, we literally construct our knowledge dynamically as we 
interact with the world. 

We observed specific instances of this kind of knowledge construction as 
we observed the students talking about how to represent abstract concepts, 
such as nitrogen, fixed nitrogen and energy from the sun. Signs that represent 
abstract concepts can be truly and completely arbitrary. And yet, students 
chose to embed meaning in the signs that they selected, through their form, 
color, or how they were transformed. For example, the students decided to 
represent free nitrogen as a yellow sphere. This in and of itself is not 
remarkable. However, when free nitrogen changes to fixed nitrogen through 
one of several processes, that same yellow sphere was augmented by four 
smaller spheres, representing the additional chemical components that are 
now a part of the nitrogen, while retaining the original symbol of the free 
nitrogen itself. Thus, the students' understanding of the nitrogen-fixing process 
as the addition of chemical components was represented by a symbol that 
acquired its own components as a result of student action in the VE. 

Another example of this embedded meaning-making took place during 
discussion about the color and form of signs for energy from the sun and water 
vapor, two components of the water cycle. The process by which water in its 
various forms moves through the wetland (and all other environments, for 
that matter) is by evaporation, condensation, and precipitation. In 
representing these three interrelated processes, students wanted a clear way 
of conveying what needed to mix with what in the environment. In discussing 
color, there was no disagreement that energy from the sun should be yellow, as 
the sun is yellow in these students' perception, and that water vapor should be 
blue, since the preponderance of water that we see on the earth appears blue 
in color. Students could clearly map the color onto the representation as a 
clue to the participant about what these arbitrary signs might represent. 
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The next topic of discussion was form- what does energy from the sun 
look like? Is it a ray? A string of little suns streaming from the big sun to the 
earth? Is it a big yellow blanket? How can we represent an object that can be 
used to create the water vapor necessary to illustrate the water cycle? Should 
the energy move towards the earth, or should the participant move it? All of 
these questions were important to the students. La the end, the symbol they 
selected for energy from the sun was a yellow bolt of energy, since they could 
also agree that what the sun provides is energy in the form of heat. This bolt 
was easy to pick up and take down to the lake, where the bolt mixed with the 
lake to generate water vapor. 

The discussion of the form water vapor should take was also revealing. 
Should it be a series of droplets? Just a dense fog? Should the water vapor 
automatically move up towards the sky once it appears? In the end, the 
students elected to create a series of three semi-transparent (since water 
vapor is essentially transparent) arrows that would appear just above the 
lake, and point up towards the sky. As with the bolt of energy, they also felt 
that it was more important that the participant be able to grab the water 
vapor, and manually move it towards the cloud, representing the process of 
condensation. 

The process by which the students, with the assistance of both their 
regular teachers and HIT Lab staff, came to these conclusions illustrated 
some important pedagogical implications of sign-creation. In traditional 
classrooms, the underlying pedagogy is based on the metaphor "student as 
container" and "teacher as knowledge bearer". This approach would certainly 
have precluded the creativity and richness inherent in the discussions about 
the students* view of what constitutes a wetland, and how to convey abstract 
information within that environment. As Cunningham (1992) points out: "We 
teach teachers those techniques that have been scientifically shown to 
efficiently and effectively communicate this knowledge to students: classroom 
organization and management, questioning techniques, lecturing, discussion, 
preparation of materials, and so forth. All are geared toward effective 
communication." (p. 168) 

In contrast, the semiotic view, to which we subscribed in the wetlands 
project, of "student as meaning maker" and "teacher as facilitator" will result 
in ail entirely different set of teaching and learning practices. In this 
paradigm, the student and the teacher work together through dialogue to 
inform one another about their perceptions and to create personal meaning. 
This approach has unlimited potential for meaning-making or knowledge 
construction. In the view of Cunningham (1992), Landow (1989), Gardner 
(1983), and Friere (1970), our job as educators is to provide a rich environment 
in which to build these structures. As specifically stated by Cunningham: 
"We have to shift to pedagogical strategies that promote a student's ability to 
see that multiple perspectives may be brought to bear on a problem; that 
coming to understand another's view requires dialogue, not simply listening; 
that learning can and often should occur in a social setting, not as some 
private act; and that learning should be situated within realistic contexts 
about which the students care or about which they have made some kind of 
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commitment." (p. 181). Based on what we discovered at Kellogg Middle School 
and in other projects where we have allowed the students free rein to create 
their VEs based on their own knowledge and imagination, we have learned the 
value of the semiotic approach to learning and pedagogy. Building VEs 
provides a great opportunity to shift the learning and teaching metaphor 
towards one with a greater emphasis on semiotics. 



World Design: The Nature and Importance of Presence. 

Real-time visual and auditory cues afforded by the technology, described 
previously, with which we create VEs, combined with a participant's ability to 
interact with objects in the virtual environment, often lead participants to 
have a strong sense that they are "in a place". They feel they are inside the 
computer simulated environment, rather than just looking at a video display. 
This psychological sensation, known as "presence", is thought to be the 
essence of virtual reality. VR researchers speculate that presence improves 
cognitive and motor performance witliin a virtual environment, and improves 
learning and transfer of training. VR reduces or removes the interface 
between the user and the computer (M. Bricken, 1991; Winn, 1993;, reducing 
the amount of attention participants must direct at developing an 
understanding of the computer interface itself. 

Despite this improvement over traditional computer interfaces, 
students in a virtual environment are still performing a divided attention task. 
Part of their attention is drawn into the virtual environment, but at some level 
the student remains aware of the fact that they are still standing in the 
la oratory or classroom wearing a bulky helmet. Feelings of presence are 
therefore likely correlated with the amount of attention drawn into the virtual 
world and away from the laboratory. Manipulations of VR designs that create 
a stronger sense of presence (see Hoffman, Prothero, and Wells, in 
preparation) likely draw more attention into the VE. 

Increasing the amount of attention paid to information can improve 
recall. For instance, emotionally arousing events typically draw more 
attention and are better remembered than emotionally neutral events (e.g., 
Christianson, Loftus, Hoffman and Loftus, 1989). (Interestingly, Laurel 
[1993] suggests that researchers think of presence as an emotion.) In order 
to maximize the quality of the human/computer interface and to increase the 
likelihood that students will pay attention and learn the information presented 
in VEs, designers should attempt to maximize the sensation of presence in 
VEs. 

This task is made easier by the fact that VR allows multimodal 
representation of objects. When considering how to represent an object in a 
way that maximizes the feeling of first-person participation and the feeling of 
presence, world builders should consider what sensory inputs will be available. 
For instance, hearing a coke can fall from the machine when buying a virtual 
pop increases subject's psychological experience that "it's the real thing 11 , 
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when the sound is appropriate and accurately located (Barfield and Hendrix, 
1995). In contrast, hearing a balloon bark wh .n punctured could conceivably 
decrease the student's sense of presence. 

The provision of tactile cues in VEs is technically difficult and 
economically cumbersome. As a result, VR systems that give tactile 
feedback are not common. In the real world, tactile cues are largely processed 
without our awareness, but make a critical contribution to our sense of self 
and our construction of reality. A number of researchers are developing 
tactile-feedback devices. For example, researchers at MFITs Research 
Institute in Tsukuba, Japan, have constructed a robot-like force-feedback 
device. Participants push a handle along a virtual wall. The robot stops the 
participant's hand in appropriate places as if a real wall were blocking motion. 
Eventually, such haptic feedback will likely become an important addition to 
commercially available virtual reality engines. 

In the meantime, Hoffman, Winn, Hollander, and Rousseau (in 
progress) are developing a cheap shortcut. In a proposed study, students in a 
"see only" condition will see a 3-D virtual image. Other students in a "see and 
feel" condition will see the same image, but will also be able to reach out and 
touch the virtual objects. For the "see and feel" condition, we simply place real 
objects (for example, a rubber ball) on a table within the participants 1 grasp in 
spatial locations matching the approximate location of the virtual object (a 
blue ball in the VE). This technique, which we call "tactile augmentation", is a 
mixture of real and virtual representations. Experiments designed to assess 
whether tactile augmentation increases the sense of presence and whether it 
enhances learning in the virtual world are presently underway. 

According to Laurel (1993, p 117), virtual reality is more than a new 
tool, it is a new communication medium: "In task-oriented applications, new 
technologies sire allowing researchers to replace indirect or symbolic 
representations and manipulations with direct, concrete ones - for example, 
physically pointing or speaking, as opposed to typing, spatial and graphical 
representations of data as opposed to textual representations. Likewise, the 
evolution of natural-language interfaces is beginning to replace the elaborate 
conventions of menu-based and command-based systems with systems that 
employ language in ways that are mimetic of real-world activities like 
conversation and question and answer". She emphasizes that VR gives 
participants (students) experiences, as opposed to information. In our terms, 
VE's create knowledge from information through experience. She states "the 
human is an indispensible ingredient of the representation, since it is only 
through the person's actions that all dimensions of the representation can be 
manifest". In light of this, one of the roles of the world designer is to create 
representations of objects and environments that provide contexts for action. 

Presence is enhanced when VR allows for changes in the behavioral 
state or representation of a sign in response to the participant's actions (i.e., 
the creation of new signs from existing signs). This "action" feature is very 
valuable. When a . tudent in Byrne's Chemistry world fills the outer orbital 
shell by placing two electrons near the nucleus, a huge bulb representing a 
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completed orbital shell appears in the world. The element of surprise at the 
metamorphosis of old signs into new can be of great educational value. Sign 
metamorphosis like this should cue subjects about what they should do next. 
For example, in the water-cycle world, touching a cloud makes rain, suggesting 
that doing something with the rain is the next step. And, as we saw above, 
correctly fixing free nitrogen changes the symbol that the student 
manipulates, suggesting that fixed nitrogen needs to be used for the next task. 

Finally, a high level of presence, which draws students' attention into the 
VE, can make it easier for them to actively figure out how the world works. 
Constructivist world designs, such as we described earlier, can encourage 
scientific thinking. The students make an educated guess about what they 
should do next, and they "test their hunch". Encouraging students to stand in a 
scientist's shoes is a successful approach to science education (see Bruer, 
1993). Rather than passively memorizing the fruits of the labor of countless 
faceless scientists, students formulate and test their own hypotheses about 
how the world works. If a high level of presence allows the results of their 
operations upon objects in a VE to be immediate and to be meaningful in the 
context the VE provides, thus confirming or disconfirming their hypothesis, 
their understanding can be improved. On the other hand, low presence allows 
the intrusion of the real world into the VE, distracting them and preventing 
them from building their own understanding. 



World Design: Embedding Guidance in VEs. 

How the behaviors of objects are represented in a VE is important to 
providing guidance to students who are having difficulty constructing 
knowledge. One of the biggest challenges facing designers of VEs for both 
education and training is how to provide assistance to participants without just 
telling them what to do. The simplest approach, which we use when necessary , 
is for an observer to give verbal hints and suggestions to a student who is 
immersed in a VE. However, as one might expect, this reduces presence as 
attention is drawn away from the VE and into the world outside. Placing an 
"avatar" into the VE with the student is another approach. The avatar 
observes the student's actions and provides guidance when necessary. While 
this approach maintains and perhaps even enhances presence, it is difficult to 
implement. A third approach is to build guidance into the very behavior of the 
VE. This section provides a brief discussion of this technique. 

As we explained in the previous section, presence, and thus learning, are 
improved if signs change their appearance in meaningfiil ways as a result of 
participant actions. However, if a student performs an unanticipated action (a 
creative one, a wrong one), the sign typically does nothing. We feel that this is 
the equivalent of a standard CAI program printing "Answer unrecognized" onto 
the screen. It certainly causes frustration and confusion. There are other 
possibilities, some of which were described by Winn & Bricken (1992) in a VE 
that obeyed the laws of Algebra rather than the laws of the Newtonian 
universe. The following example is taken from their article. 
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If, during the solution of an algebra problem, a term is moved from one 
side of the equals sign to the other, the term's sign changes from positive to 
negative or vice versa. Assume that this action ik symbolized by inverting the 
symbol used to represent the term, in this case a colored block. One of three 
things could happen when a student moves a block across an equals sign 
without inverting it. First, the VE might allow the student to make this error. 
In this case, the solution to the problem is "bugged" and, ojice the student 
realizes subsequently an error has been made, the steps in the solution must 
be traced back to where the bug occurred. In short, no guidance at all requires 
a student in a VE to engage in trouble-shooting. Second, the VE might make 
the block float back to its original position. This is what happens to misplaced 
atomic particles in Byrne's Chemistry World (Byrne et al. 1995). In this case, 
the student knows an error has been made, but is not told what it is. The 
student repeats the step using a different action. In the third possibility, the 
block inverts itself and settles into place. The student knows an error was 
made and also what the correct action should have been. The VE is self- 
correcting. 

There is nothing startling about this example. The various behaviors of 
the algebraic term correspond to fairly standard practices of providing 
feedback to students. However, the ways of representing the behavior of 
objects in VEs is, within the limits of the technology, unlimited. Because 
objects in VEs are themselves signs, they can be programmed to behave in 
any way at all. This means that in VEs such as the Algebra world, whose 
purpose is more didactic than constructivist, pedagogy can be built seamlessly 
into the environment itself. Thus, presence can be maintained while students 
get guidance on specific actions they must perform. In VEs designed to permit 
students more freedom to construct knowledge for themselves, imagination 
could be stimulated by introducing a measure of randomness into behaviors. In 
cases where VEs are built by students themselves, and where the activity of 
building a VE rather than visiting one is how learning is encouraged, students* 
experimenting with how to represent procedures and principles as behaviors is 
an act of symbol-creation and meaning-making that serves a similar purpose 
as deciding how to represent each object. And like deciding how to represent 
objects, chosing and experimenting with behaviors that act as symbols for 
abstract procedures and principles, like fixing nitrogen or changing electron 
spin, helps students construct knowledge from information. 



Conclusion. 

It is premature to draw hard and fast conclusions about the 
effectiveness of either building or visiting VEs for knowledge construction by 
students. The technology is still too young and the research too fragmented. 
However, we do have evidence that visiting a VE improved students* 
understanding of basic concepts in chemistry (Byrne et al. 1995) and can 
report here that the world-building project at Kellogg Middle School produced 
significant increases in students* understanding of wetlands ecology. We also 
know, from the projects described here and from others, that students have 
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little difficulty with the software used for building worlds (Bricken & Byrne, 
1993), that they enjoy doing so and can do so in a collaborative manner 
(Osberg, 1994a). 

It is also somewhat premature to conclude that semiotic theory is the 
best tool by means of which to conceptualize, study and discuss learning in 
VEs. In this paper, we have made a first attempt to explain where we think 
the most fhiitfiil links between semiotics and VEs are likely to be found. Still, 
the fact that people who design, build and visit VEs are dealing with signs that 
represent reality in some impoverished way is far more self-evident than it is, 
perhaps, for those working in similar ways with "real" environments. This 
means that, within our conceptual framework, it is both necessary and useful 
to think of transforming data into information as the creation of signs, whether 
this action is performed by VE designers, students themselves, or computer- 
driven transducers. 

The construction of knowledge from information has many of the 
characteristics of "semiosis" (Cunningham, 1992) or meaning-making. We 
believe that, in the case of learning in VEs, presence must be established at a 
high level for knowledge construction to be successful. The more a student is 
drawn into a VE by the ability to interact with it naturally, by the conviction of 
being in another place and by the way objects and their behaviors are 
represented, the more likely it seems that the student will construct personal 
and meaningful knowledge of the world. However, our research agenda that will 
confirm or refute this assumption is still in its early stages. 

And what of wisdom? We certainly hope that what our students acquire 
by working with us is "a virtue" and that it provides "a guide for action" 
(Sternberg, 1990). It may be that the actions of physically building and 
interacting with representations of objects may accelerate students' ability to 
act on the knowledge they have constructed. However, we are still persuaded 
that the acquisition of wisdom takes a long time. This suggests the need for 
longitudinal studies of students who have enjoyed extended experiences in VEs. 
The scope of any such studies would be immense, implying, perhaps, that the 
question of wisdom can only be answered when we know whether VEs improve 
or impoverish our quality of life in general. In short, only time will tell. 
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